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Abstract 

The drill-and-practice mathematics program run in Mississippi and 
California showed positive results for grades 1 through 6 in Mississippi 
and for some grades in California in comparison with matched control groups. 
The tutorial mathematics program run with first and second graders in a 
deprived area in California had a statistically significant positive effect 
only for slow learners in grade 1 in comparison with traditional classroom 
instruction. In a Russian course for college students^ CAI students per- 
formed significantly better than control students on final examinations for 
two of the three academic quarters for the first-year course and on the 
only quarter examination evaluated thus far in the second-year course. 
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Although computer-assisted instruction has reached the operational 
stage in various parts of the curriculums in a number of places in the 
United States, very few "hard data" evaluations of student achievement in 
these programs have yet been published,, The purpose of this paper is to 
report the results of the evaluative testing of students in three programs 
that have been running at Stanford over the past several years. The first 
is the drill- and -practice program in elementary mathematics. Results are 
reported for schools in California for the 1966-67 and 1967-68 academic 
years and for schools in McComb, Mississippi for the 1967-68 academic year. 

The second program is a first- and second-grade tutorial mathematics curriculum 
conducted at the Brentwood School in East Palo Alto, California. Results of 
evaluative testing for 1966-67 and 1967-68 are given. The third program 
is the tutorial curriculum in elementary Russian at Stanford University, 
which is in its second year of operation at the time of writing this paper. 

We shall not attempt a wide-ranging evaluation of computer- assisted 
instruction in terms of reporting observations of student behavior; the 
results of student, parent or teacher questionnaires; or detailed analyses 
of curriculum performance. Some results of this kind have already been 
published in Suppes, Jerman, and Brian (1968). We also shall not report 
evaluation of the Stanford tutorial programs in reading. For this the 
reader is referred to Atkinson (1968). The purpose of the present paper 
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is to concentrate on the classical comparison of experimental groups with 
control groups and to compare their relative rates of achievement. In the 
case of the two mathematics programs, the primary instruments of evaluation 
were Stanford Achievement Tests (Kelley, Madden, Gardner, & Rudman, 1963); 
which are not a product of Stanford University hut are commercial tests. 

In the tutorial mathematics program, individualized Stanford-Binet 
intelligence tests were used. In the Russian program, which was under 
the direct supervision of Professor Joseph Van Campen at Stanford University, 
the evaluation consisted of comparative performance on midterm and final 
examinations in the course. 

It should he emphasized that the primary purpose of this paper is 
to present without extensive interpretation the evaluative results. We 
do conclude with some discussion of the results, hut the main function 
of the paper is to present in standard data form the results of the testing. 

1. Drill - and - practice Program in Elementary - school Mathematics 

The drill-and-practice program began in the spring of 1965 with 4l 
fourth-grade children who were given daily arithmetic drills on a teletype 
machine in their classroom. By the end of the 1965-66 school year, 

270 students in grades 3 through 6 in three California elementary schools 
were participating in the program. For a detailed account of 1965-66, see 
Suppes, Jerman, and Brian (1968). During the 1966-67 school year, the pro- 
gram was further expanded to include grades 1 through 6 with more than 
1,500 students involved. Student participation increased again during 
1967-68 with approximately 1,000 students in California, 600 students 
in Mississippi, and 1,100 students in Kentucky. 







Because changes occurred in the curriculum and the computer system 
as the program developed during the first two years, statistical evaluation 
was not begun until the 1966-67 academic year. During 1966-67 and 1 967-68, 
Stanford Achievement Tests were used for evaluation*, The primary aim of 
the program was to provide drill and practice in the skills of arithmetic, 
especially computation, as an essential supplement to regular classroom 
instruction*, The concepts presented to the students for drill and review 
at the computer terminal have been previously introduced in the classroom 
by the teacher. 

For the 1966-67 and 1967-68 school years, the curriculum material, 
for each of grades 1 through 6, was arranged sequentially in blocks to 
coincide approximately with the development of mathematical concepts intro- 
duced in several text series. There were 20 to 27 concept blocks for each 
grade level. Each concept block included a pretest, five days of drill, 
a posttest, and sets of review drills and review posttests, A brief de*- 
scription of the material in each concept block is shown in Table 1, 

Insert Table 1 about here 



Parallel forms of a test were prepared for each concept block. The 
test consisted of an equal number of problems from each of five levels 
of difficulty. For a given student, different forms of the test were 
assigned for the pretest and for the posttest in each block. The form 
assigned for the pretest was counterbalanced among students. The forms 
of the test not assigned as a pretest or a posttest for a given student 
were divided into halves and used as review posttests for that student. 
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TABLE 1 

Concept Blocks for Grades 1 -6, Drill-and-Practice Program 1966-67 



Grade 1 


Grade 2 


Block 


Description 


Block 


Description 


1. 


Counting, How many, 0-9 


1. 


Addition facts to 10 , horizontal 


2o 


Counting in sequence 


2, 


Subtraction facts to 10 , 


3 o 


Sums to 4 




horizontal 


4 , 


Sums to 4 , vertical, mixed 


3 . 


Addition and subtraction facts 
to 10, vertical 


5 . 


Differences to 4 , vertical, mixed 


4 , 


Addition facts to 10 , mixed 


60 


Sums to 6, vertical, mixed 




horizontal and vertical 


7 ° 


Sums to 7, vertical, mixed 




with variables 


80 


Differences to 7, vertical, mixed 


5 . 


Mixed addition and subtraction 
to 10, mixed horizontal and 


9 ° 


Sums to 9, vertical, mixed 




vertical 


10 0 


Sums to 10, vertical only 


6. 


Counting by l’s and 2 ’s; finding 


11 0 


Differences to 10, vertical only 




what comes before and after 


12 0 


Sums to 10 with variables 


7 . 


Addition, 11 , 12 , 13 , horizontal 
and vertical 


13 . 


Differences to 10 with variables 


8. 


Subtraction, 11, 12 , 13 , 


14 . 


Sums and differences to 10, 




horizontal and vertical 




horizontal 


9 - 


Mixed addition and subtraction, 


15 0 


Sums and differences to 10, 




horizontal and vertical to 13 


16 . 


vertical format 


10. 


Units of measure; counting; 


Sums and differences to 10 




inequalities 




with variables 


11. 


Addition, 14 , 15, 1 6, horizontal 


17 0 


Sums to 10, 3~digit numbers 




and vertical 


18, 


Column addition, sums with 10 ’ s, 


12. 


Subtraction, 14 , 15, 16 , 




no regrouping 




horizontal and vertical 


19 . 


Column subtraction, no regrouping 


13 . 


Mixed addition and subtraction, 


20, 


Mixed addition and subtraction 




horizontal and vertical, 14 , 15, 16 




in columns, facts to 10 


14 . 


Word problems; units of measure; 


21, 


Mixed addition and subtraction, 




counting to 200’ 




inequalities 


15 . 


Fractions, 1/2, l/ 3 , l /4 


22, 


Mixed 1- and 2-digit column 


16. 


Addition, 17 , l8, 19, horizontal 




addition and subtraction 




and vertical 


23 . 


Sums to 10 with form 


17 . 


Subtraction, 17 , 18 , 19, 




a + b = c + d 




horizontal and vertical 


0 

- 3 * 

OJ 


Sums to 10 with variables, 


18. 


Mixed addition and subtraction, 




form a + b = c + d 




horizontal and vertical 




(continued) 




(continued) 
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Table 1 (continued) 




Grade 1 (continued) 




Grade 2 (continued) 


Block 


Description 


Block 


Description 1 



25 . 
27. 



Special addition and subtraction 
Special mixed drills 



19 » Units of measure; counting; 
inequalities 

20 <, Multiplication, 2*s and 5’s to 9 
(i.e. , 2x0= 000 

2x9 = ) 

21 o C and A Laws for addition, 

subtraction, and multiplication 

22. Achievement Tests 

25 . Mixed drill: fractions; units 

of measure; inequalities; 
multiplication 

2k o Mixed drill: addition, 

subtraction, multiplication 

25 '. Special addition and subtraction 

27. Special mixed drills 



Table 1 (continued) 



Block 



Grade 3 
Description 



1 1 Mixed addition and subtraction, 
horizontal format, sums 0-l8 



Block 



Grade 4 
Description 



Addition, 1 and 2 digit, 
vertical and horizontal 



2 . 



Addition, sums 0-l8, horizontal 
and vertical 



2 . 



Subtraction, 1 and 2 digit, 
vertical and horizontal 



3* 



Subtraction, sums 0-l8, 
horizontal and vertical 



3- 



Subtraction, 2 and 3 digit, 
vertical format 



4. 



5* 

6, 

7 

8, 

9. 



10 . 



ll. 

12 o 



Addition, no carry, vertical 
(2 addends, 3 digit) and 
(3 addends, 2 digit) 

Subtraction, no borrow, vertical 
2 and 3 digit 

Addition, vertical with carry 

Subtraction, with borrow 

Mixed addition and subtraction, 
carry and borrow 

Measure and word problems and 
inequalities 

Column addition and subtraction; 
add; subtract 

Measure, inequalities 

Multiplication, horizontal, 

2 ’s and 3 ’s 



13. 



14, 



15 o 



Mixed multiplication and 
division, 2 's and 3 J s 

Division, ladder form 
1 digit into 2 digit 

CAD Laws, add 
subt o 
mult. 



16. Mixed drill: measure, word 

problems, inequalities 

17, Fractions 



I80 



Multiplication, horizontal 
2* s 9's 



19 0 Mixed drill: multiplication, 

division, fractions 

20. Division, ladder form 
1 digit into 3 digit 

(continued) 



4. 

5* 

6 . 



7° 



8 . 



9. 



Addition, 2 and 3 digit, 
column addition 

Mixed addition and subtraction, . 
vertical format, limits same as 
Blocks 3 and 4; word problems 

Measure: length, time 

Weight, money: some word 

problems 

Multiplication, 2’s -)9’s, 
horizontal format; levels by 
products 

Mixed addition, subtraction, 
and multiplication; word problems; 
Add. and subt., vertical format; 
limits same as Blocks J), k, and 7 
Multiplication, horizontal format 

CAD Laws: days 1-4 apply law 

day 5 identify law 



10 . 



11 . 



12 . 



Division: ladder form, no 

remainders, level by products, 
single-digit divisor, 2 's -* 9 ’s 

Multiplication: 2’s through 12* s 

horizontal format, level by 
products 

Fractions: identify (to l/ 8 ) 

simple reducing 



13. Mixed drill: mult,, div,, 

fract., inequalities, word 
problems; same limits as 
Blocks 10, 11, 12 (horizontal 
and vertical) 

14, Long division: ladder form, 

1-digit divisor, 2-4 digit 
dividend, random divisors 

(continued) 
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Table 1 (continued) 



Grade 3 (continued) 


Grade 4 (continued) 


Block 


; Description 


Block 


Description 


21. 


Multiplication, vertical 
1x2 digit 


15. 


CAD Laws: days 1-3 using law 

days 4-5 identify law 


22. 

23. 

24. 
25* 

27. 


Achievement Tests 

Mixed drill: column add} subt.j 

multiply 

CAD Laws 

Special addition and subtraction 
drills 

Special mixed drills 

i 

i 

t 


16 . 

17. 

18 . 
19. 


Fractions: addition, subtraction, 

reducing 

Measure: time, money, liquid 

measure, length, weight; 
some word problems 

Multiplication: multiples 

of 10, inequalities 

Mixed drill: mult., div., frac. 

CAD Laws; same limits as Blocks 
14, 15, 16 , 18 ; some word 
problems 






20. 


Long division: ladder form, 

1-digit divisor, 2-4 digit 
dividend, random remainders 






21. 


Fractions 






22. 


Achievement Tests 






23. 


Mixed drill: long division, 
fractions, negative numbers; 
same limits as 20, 21, 22 






2k). 


Estimation of quotients in 
division 






25. 


Special addition and multiplication 
drills 






PO 

CT\ 

• 


Special subtraction and division 
drills 






27 . 


Special mixed drills 
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Table 1 (continued) 





Grade 5 




Grade 6 


Block 


Description 


Block 


Description 


1 . 


Addition, vertical and horizontal 


1 . 


Mixed drill: l/2 column, add. , 




1 , 2 , and 3 digit 

Level 4, carry to 10* s 

Level 5 , carry to 10’ s or 100's 


2 , 


subt.; l /2 multiplication; some 
involving decimals 

Multiplication: 2 's ->12's. 


2 * 


Subtraction, vertical and 




level by products, horizontal 


3. 


horizontal, 1 and 2 digits 
Mixed addition and subtraction, 


3- 


format 

Column multiplication: 


1 4. 

i 

* 


3 , 4 digit; mixed borrow, carry 

Multiplication, 2's ->12's 
Level by products, horizontal 


4. 


(l digit) x (2 digit) through 
(2 digit) x (3 digit) 

Division: ladder form, 


5* 


Multiplication, vertical 
Up to 1 x 4 digit 


5* 


1 -digit divisor 

Fractions: factors, reducing, 


6 . 


Carry, no carry 

Mixed drill: multiplication, 


6 . 


comparing, simple add., subt. 
Mixed drill: inequalities. 


7 - 


division, fractions 
Division: ladder form 




decimals, word problems, 
exponents, add,, subt., mult., 
div. 


8 . 


Level 3- 1 into 3 digit 

Measure 


7- 


Division: ladder form to 2-digit 

divisors 


9* 


Multiplication, vertical 


8 . 


Fractions: add, subtract 


H 

O 

o 


2 digit, 2 's -* 12 ' s 
Mixed drill: column add. 


9. 


Measure: length, time, money, 

temperature, liquid measure 




subtraction, multiplication, 
decimals, CAD Laws 


10 , 


Ratio: per cent 


11 0 


Division: ladder format 


li. 


Division: ladder form, 2-digit 


12 . 


Level 3* 2 into 3 or 4 digit 

Fractions 


12 . 


divisor 

Mixed drill: fractions (add., 


13- 


Measure, decimals 

✓ 




subt., mult,), ratio, per cent, 
div., decimals (add,, subt,) 


14. 


CAD Laws 


13- 


Fractions: decimal, add,, 


15. 


Division: ladder format 




subt . , mult . 


16 , 


Fractions 


14. 


CAD Laws: days 1-4 apply law 


17- 


Mixed drill: multiplication, 

division, fractions 


15- 


day 5 identify law 
Multiplication: multiples of 10, 


18 . 

19- 


Measure 

Fractions, decimals 


1 6 . 


horizontal format 

Division: ladder form, 2-digit 

divisors, 3"5"<3-igit dividends 


no 

o 

& 


Mixed drill: multiplication, 


17- 


Mixed drill: fractions (+, -, 




division, decimals 
(continued) 


x, *•) , fractions (column add.) 
CAD Laws, division 

(continued) 
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Table 1 (continued) 





Grade 5 (continued) 


Grade 6 (continued) 


Block 


Description 


Block I 


j Description 



21 . 

22 . 

23 . 

2k. 

25 . 

26. 
27. 



Division, ladder format 

Achievement Tests 

Mixed Drill: summary 

Estimation of quotients in 
division 

Special addition and multiplication 
drills 

Special subtraction and division 
drills 

Special mixed drills 



18. I Measures: all, including a 

I few metric; area, volume 

19* j Ratio, per cent 

20. I Mixed drill: all operations, 

I per cent, decimal mult. 

21. Negative numbers: add, subtract, 

I multiply 

22. I Achievement Tests 

23. I Mixed drill: summary 

2k. I Estimation of quotients in 
I division 

25. I Special addition and 

I multiplication drills 

26. I Special subtraction and 
I division drills 

27. I Special mixed drills 
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For each day of drill, five drills, one at each of the five level > of 
difficulty, were prepared: a total of 25 drills per block# Several sots 
of review drills for each block were also prepared at the five defined 
levels of difficulty# The drills and the review drills for the most dif- 
ficult level, level 5j nnd the level 5 problems on the tests were rewritten 
for the 1967-68 school year at a higher difficulty level than those presented 
in the 1966-67 curriculum. 

Each student responded to problems presented on a teletype located in 
the school. The Model-35 teletypes were connected to the PDP-1 ccmputer 
at Stanford via telephone lines# After the student signed into the program 
by typing his assigned student number and his first name, the teletype 
printed his last name and presented the appropriate set of problems. The 
temporal pace of the problem presentation was determined by the student. 

The materials presented to the student for the seven days required 
for each concept block were: 



Day 1 


pretest; 


Days 2-5 


drill and review drill; 


Day 6 


drill and review posttest; 


Day 7 


posttest 0 



Examples of the format for several types of problems are shown in Figure 1. 

Insert Figure 1 about here 



The teletype printed each individual problem and then positioned itself 
to accept the answer in the appropriate place. The student typed in the 
answer. If his answer was correct, he proceeded to the next problem. 



10 



GRADE 


BLOCK PROBLEM 


1 


1 HOW MANY MS... 

R M R R M M 
M R M M R M 


1 


2 COUNT. 

10 11 13 


1 


4 3 ♦ 1 * 


2 


4 9 ♦ 1 * 5 ♦ 


2 


5 7 ♦ N * 9 

N * 


2 


9 1 1 

♦ 2 


2 


9 1 0 

- 3 


3 


1 ♦ 35 * 38 


3 


4 2 3 

1 4 

♦2 1 


4 


2 3 6 

' *2 ..3 


4 


6 3 YD. AND 2 FT. « — FT. 


4 


9 36 X <28 ♦ 34) * C X 28) ♦ < X 34) 


5 


4 X 11 * 33 


5 


5 2 9 4 

... ,X4 


5 


6 1/3 OF 18 « 


6 


4 5/95 


6 


5 TYPE THE MISSING NUMERATOR OR DENOMINATOR. 

2/3 /9 


6 


6 TYPE « OR * OR > 

3 + 8 9*4 


6 


7 <17 X ) ♦ 9 • 28722 



Fig e l e Samples of problem formats for grades 1 through 6, 
drlll-and-practice program,. 
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If he input the wrong answer , the teletype printed NO, TRY AGAIN and presented 
the problem again. If he made a second error , the teletype printed NO, THE 
ANSWER IS . ». and presented the problem once more. If the student input 
the wrong answer for the third time, he was given the correct answer and 
the teletype automatically proceeded to the next problem. The student was 
allowed from 10 to ^-0 seconds to respond, depending upon the type of problem 
presented. If a student took more than the allotted time to input his answer, 
the procedure just described was followed, but the teletype printed TIME IS 
UP, TRY AGAIN in place of NO, TRY AGAIN . 

The level of difficulty of the first day of drill was determined by 
the student’s performance on the pretest according to the criteria presented 
in Table 2. The level of difficulty of each successive drill in the same 
concept clock was determined by the student ’ s performance level on the 
preceding day’s drill. Thus, if the student’s performance on a drill was 
80 per cent or greater his next drill was one difficulty level higher. 

A score of less than 60 per cent branched him down a level for the next 
drill. Otherwise, the student remained at the same^-diff iculty level for 
the next drill. 

Whereas the drill content was the same for all students in a class 
with only the difficulty level changing as a function of the preceding 
day s performance, the content of the review drills differed among students 
as a function of the total past performance history of each student. The 
computer individually selected the review drills to correspond to the content 
of the past block that had the lowest posttest score for that student, with 
the restriction that he was not reviewed for two seven- day blocks in a row 
on the same past clock. The level of difficulty of the review drills was 




determined by the posttest according to the criteria presented in Table 2; 



Insert Table 2 about here 

the difficulty level remained constant for all four days of review. Once 
a student had received a set of review drills on a given concept block, 
the score on the review posttest, given on the sixth day, replaced the 
previous posttest on that concept block for determining the concept block 
and difficulty level for future review drills. 

The branching structure for a seven-day sequence of problems is shown 
in Figure 2, Each darkened circle represents a drill; each open circle 

Insert Figure 2 about here 

represents a review drill. To make up for absences, a student could take 
more than one drill per day, branching accordingly after each drill. 

Evaluation 

To evaluate the effectiveness of the drill-and-practice program, 
the arithmetic portion of the Stanford Achievement Test (SAT) was ad- 
ministered to both experimental and control classes. Four different 
levels of the SAT were used. Each level had one, two, or three arithmetic 
sections which are described briefly in Table 3* Unless otherwise noted, 

Insert Table 3 about here 

the tests were administered in October and again in May by either the 
classroom teacher or a member of the staff at the Institute. 
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TABLE 2 

Branching Criteria 
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From Pretest to Drill 



From Drill to Drill 



* 

f 

\ 

\ 


Per cent 
correct 


Level assigned 
for dr ill 

1 

1 


Per cent 
correct 
on drill D. 

l 


Level assigned 

for drill D. , n 
l+l 


1 

S’- 


0 - 


19 


1 


o - 59 


next 


lower level 


I 

V 


20 - 


39 


2 ; 

i 


60 - 79 


same 


level as D. 

l 


| 


40 - 


59 


} 

3 I 

l 


80 - 100 


next 


higher level 




60 - 


79 


4 | 








\ 


80 - 


100 


5 i 









Also from Posttest to Review 
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Block with lowest posttest performance 




TABLE 3 



Description of Stanford Achievement Test Batteries - Arithmetic Portion 

Primary I: Middle of grade 1 to middle of grade 2 

A. Arithmetic, 63 items 

(Measures, problem solving, number concepts) 

Primary II: Middle of grade 2 to end of grade 3 

A. Arithmetic Computation, 60 items 

(Addition, subtraction, multiplication, division) 

B. Arithmetic Concepts, 46 items 
(Numbers, measures, problem solving) 

Intermediate I: Beginning of grade 4 to middle of grade 5 

A, Arithmetic Computation, 39 items 

(Addition, subtraction, multiplication, division) 

Bo Arithmetic Concepts, 32 items 

(Place value, meanings and interrelationships of operations, 
average, per cent, etc.) 

C. Arithmetic Applications, 33 items 

(Reasoning and problem solving in area, ratio, volume, 
averages, graphs, etc.) 

Intermediate II: Middle of grade 5 to end of grade 6 

A. Arithmetic Computation, 39 items 

(Addition, subtraction, multiplication, division) 

B. Arithmetic Concepts, 32 items 

(Place value, meanings and interrelationships of operations, 
average, per cent, etc.) 

C. Arithmetic Applications, 39 items 

(Reasoning and problem solving in area, ratio, volume, 
graphs, averages, etc.) 



1.6 




Although the publishers of SAT recommend that the test in May for 
grades 2 and 5 be one level higher than the test given in October (Table 3), 
whenever possible we administered different forms of the same test within 
a given grade; i«e., Primary II in grade 2 and Intermediate II in grade 5. 
However, when the administration of the SAT was an integral part of a testing 
program within a school system requiring adherence to the SAT manual, we 
were unable to dictate which tests or forms were administered. This fact 
must be kept in mind when deviations from standard experimental design occur. 

California 1966-67 

Tests were administered to four schools for the 1966-67 evaluation. 

Grades 3 through 6 were tested in Experimental School A and Control School B; 
grades 4 through 6 were tested in Experimental School C and Control School D. 
In each case, the control school was located in the same district as the 
experimental school. The pretest and posttest administered was Primary II 
for grade 3, Intermediate I for grade 4 , and Intermediate II for grade 6 
in all four schools. For the fifth grade, Schools A and B administered the 
Intermediate II test for both the pretest and posttest; Schools C and D 
administered the Intermediate I as a pretest and Intermediate II as a posttest. 

The difference between the posttest and pretest grade placement on the 
SAT Computation Section for each grade for School A versus School B and for 
School C versus School D was examined. The statistical results of t-tests 
and the average pretest and posttest grade placement are shown in Table 4 . 

Insert Table 4 about here 
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The increase in performance level for students in the experimental school 
was significantly greater than in the control schools for grade 3 i* 1 
School A and for grades 4 and 6 in School C. 

At the end of the school year we learned that after examining the 
results of the pretesting at their school the teachers and administrators 
at School B, a control school, instituted an additional 25 minutes per day 
of classroom instruction and practice in arithmetic for grades 4 and 5* 

Since the performance increase of the students in Control School B in grades 
4 and 5 was significantly greater than in Experimental School A, we would 
conclude that 25 extra minutes of classroom drill can be more beneficial 
than 5 to 8 minutes per day of computer-based drill. The greater gains 
for Control School B compared with Control School D supports the conclusion 
that the performance of the students in School B was a function of the extra 
drill in the classroom. The effect of classroom drill, however, does not 
detract from the effectiveness of the drill-and-practice program. Whereas 
the classroom approach required 25 more minutes of the teacher* s time 
devoted to arithmetic and 25 minutes less of the student’s time devoted 
to nonmathematical topics, the drill-and-practice program required no extra 
time from the teacher and the student lost only 5 to 8 minutes from non- 
mathematical subjects. 

Since the SAT Concepts and Applications Sections do not include many 
items contained in the drill-and-practice curriculum, they are not as 
appropriate as the Computation Section for testing the effectiveness of 
the program. However, the increase in performance level for the students 
in the experimental schools was significantly greater than in the control 
schools on the Concepts Section for grade 6, t(ll8) = 2.18, p < .01, in 
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School A; and for grade 4, t(l22) = 2.37, P < .01, and grade 5, t(l38) = 4.21, 
P ^ *01, in School C„ On the Applications Section of the SAT the experimental 
classes performed significantly better than the control classes in grade 4, 
t(l22) = 1.96, p < .05, and in grade 5, t(l38) = 2.50, p < .01, in School C. 

California 1967-68 

The administration of the SAT for evaluation of the 1967-68 program 
included grades 1 through 6 in seven different schools. Two of these 
schools included both experimental and control students, two included only 
experimental students, and three included only control students. Within 
the experimental group from 5 to 9 classes were tested at each grade level; 
within the control group, from 6 to 14 classes were tested at each grade 
level. Although the testing program for 1967-68 was more advantageous than 
that for 1966-67, in terms of number of students tested, the distribution 
of students among schools and classes made it impossible to conduct matched 
comparisons as was done in the 1966-67 evaluation. The test level adminis- 
tered as the pretest in October and as the posttest in May was Primary I 
for grade 1, Primary II for grades 2 and 3, Intermediate I for grade 4, 
and Intermediate II for grades 5 and 6. 

Again a t-test was used to determine the difference between the ex- 
perimental and control students in terns of change in performance from 
October to May within each grade. The results of this comparison and the 
average pretest and posttest grade placement are shown in Table 5 . The 

Insert Table 5 about here 
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students receiving computer-based drill and practice had a significantly 
greater increase in performance level than the control students on the SAT 
Computation Section in grades 2, 3, and 5, 

The performance of students in the experimental group was significantly 
better than that for students in the control group on the Concepts Section 
for grade 3> t(344) = 4,13, P < *01, and on the Applications Section for 
grade 6, t(399) = 2.14, p < ,05. 

Mississippi 1967-68 

The administration of the SAT for evaluation of the first year of 
drill and practice in Mississippi included grades 1 through 6 in 12 different 
schools. Eight of these schools included both experimental and control 
students, three included only experimental students, and one included only 
control students. Within the experimental group from 1 to 10 classes were 
tested at each grade level; within the control group, from 2 to 6 classes 
were tested at each grade level. 

The testing of the students in Mississippi was not as consistent, in 
terms of experimental design, as the testing in California. On the other 
hand, the computer-assisted instruction program itself operated in an 
environment that was far removed from the Stanford group responsible for 
the program, so that the lack of detailed control of the testing by the 
Stanford research group was somewhat compensated for by the independent 
environment in which the program was tested. The Primary I level of the 
SAT was administered as a pretest to students in the first grade in February 
rather than October,, The posttest was given in May. For the remaining 
grades, the pretest was given in October and the posttest in May. For the 
second grade, the Primary I level was given as a pretest and the Primary II 
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as the posttest. For grades 3 and 4 the pretest and posttest were the 
same with Primary II for grade 3 and Intermediate I for grade 4. For 
grade 5 all of the control students and two classes of experimental students 
received the Intermediate I level for a pretest and the Intermediate II 
level for a posttest; one experimental class was administered the Inter- 
mediate I test for both pre- and posttest . Although both the pretest and 
the posttest for grade 6 were at the Intermediate II level, two of the ten 
classes in the experimental group and one of the six classes in the control 
group received the same, rather than different, form for the two testing 
sessions . 

The t-value and the average pretest and posttest grade placement for 
each grade are shown in Table 6. The performance of the experimental 

Insert Table 6 about here 



students improved significantly more than that of the control students in 
all six grades. The difference between the experimental group and the con- 
trol group was largest in grade 1 where, in only three months, the average 
increase in grade placement for experimental students was 1.14 as compared 
with .2 6 for control students. 

The performance of students in the experimental group was significantly 
better than that of the students in the control group on the Concepts 
Section for grade t(76) = 3*01, P < .01, and grade 6, t(433) = 3*74, 
p < .01, and on the Applications Section for grade 6, t(433) = 4.09, 
p < .01. In grade 4, the control group improved more than the experimental 
group on the Concepts Section, t(l3l) = -2.25, P < *05. 
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Comparison between California and Mississippi 



The average grade placement on the pretest for grades 1 through 3 
was similar for the California and Mississippi experimental groups 
(Tables 5 and 6). The difference between the two groups increased in grades 
4 through 6 with the Mississippi students performing at a lower level than 
the California students. In spite of the significant gains by the experi- 
mental groups compared with the control groups in Mississippi, the dis- 
crepancy in grade placement between the California and Mississippi experi- 
mental groups for grades 4 through 6 was larger on the posttest than on 
the pretest. Thus, the overall superiority of the experimental program 
in Mississippi was related more to a lesser increase in performance level 
for the control schools in Mississippi than to a greater change in perform- 
ance level for the Mississippi experimental groups compared with the Cali- 
fornia experimental groups. In this connection it is important to emphasize 
that the California schools were all located in relatively affluent middle- 
class neighborhoods, and the average family income and educational level 
was undoubtedly higher, although we did not collect systematic data on 
economic and social variables, 

2® Brentwood Tutorial M athematics Program 

This project was designed to explore the feasiblility of using tutorial 
computer-assisted instruction in mathematics as an integral part of the 
elementary- school program. During the 1966-67 school year, 45 students 
in the first grade participated in the program; during the 1967-68 school 
year, 75 students in the second grade participated; of the 75 second 
graders, 21 had been in the first-grade program. For the 1966-67 group, 
Stanford- Binet tests were given in the fall and the SAT was given in the 
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spring to the 45 first graders active in the mathematics program and to 
the 48 remaining first graders in the same school who were participating 
in the CAI reading program. For the 1967-68 evaluation, the SAT was 
administered in the fall and in the spring to 66 of the second graders 
at Brentwood and to 8l second graders at a control school in the same 
district. (For a detailed analysis of individual student performance in 
the curriculum itself, see Suppes and Morningstar., 1969. ). 

The Brentwood system was tutorial with the computerized aspect of the 
learning of elementary mathematics completely integrated with classroom 
work. A member of our staff taught all mathematics not presented in the.', 
computer program. Curriculum material was presented by audio and visual 
displays; the student responded on a standard keyboard or used a light 
pen to touch one of the answer choices displayed on the cathode-ray tube 
(CRT). (The computer system at Brentwood was an IBM 1500. ) 

The curriculum for the first graders contained 400 lessons covering 
the t epics of counting, numerals, addition, subtraction, linear measure, 
sets and set notation, and geometry. The content and scope of the cur- 
riculum were drawn largely from Sets and Numbers , Book 1 (Suppes, 1965), 
with the addition of some topics such as oral story problems that cannot, 
by their nature, be adapted to a textbook format. An outline of the 
programmed curriculum is shown in Table 7* 

Insert Table 7 about here 



Since the programmed lessons were tutorial, many of the lessons relied 
on oral explanations synchronized with changing visual displays. The 
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TABLE 7 

Curriculum Outline for Grade 1 - Brentwood Tutorial Mathematics Program 





Number 


of Lessons 






Book 


Core 

curriculum 


Remedial 

branches 


Drills 


Description 


1 


8 


1 


0 


Using the machine 


2 k 


7 


5 


0 


Introduction to sets 


2B 


3 


3 


0 


Matching equal sets 


3 A 


5 


4 


0 


Union of two sets with one member 


3 B 


3 


1 


2 


Union of empty sets 


4 


11 


4 


1 


Geometry - learning to identify squares, 
circles, triangles, and line segments 


5 


7 


6 


0 


Balancing set equations 


6a 


8 


7 


0 


Introducing the numerals 0, 1, and 2 


6b 


9 


8 


1 


Introducing N notation with equivalent numerals 
Introducing the numerals 5 and 4 


7 A 


5 


5 


1 


Sums with N notation ( 0 - 4 ) 


7 B 


5 


0 


0 


Sms with numerals . Keyboard responses. ( 0 - 4 ) 


8 


8 


2 


0 


Review 


9 


8 


7 


1 


Addition 


10 


9 


2 


1 


Geometry 


14 A 


4 


0 


1 


Measuring line segments 


14 b 


4 


0 


1 


Concave figures, meaning of "half" 


15 


6 


7 


1 


Balancing addition equations 0-9 I 


16 


7 


5 


1 


Number words, one- six 1 


17 A 


6 

> 

* 

l 


5 


2 


Number words, zero-ten; sms to nine I 

with three addends 1 


17 B 


. * 
9 


1 


1 


Subtraction 1 


18 A 


11 


6 


0 


Subtraction combinations through six I 


18 B 


6 


2 


2 


Relating addition and subtraction a 


i9A 


12 


1 


2 


Geometry, matching similar figures ] I 


19 B 


11 


3 


1 


Subsets ] |j 


20 A 


4 


3 


2 


Review 31 


20 B 


6 


3 


2 


Subtraction c - a = b, 7<c<9 11 


21 A 


11 


0 


0 


Counting and typing to 19 1 


21B 


6 


2 


0 


Place value fl 


22 


17 


6 


1 


Addition on the nmber line 1 


23 


14 


l 


1 


Counting by tens 9 


2 4 


19 


2 


2 


Counting by fives 1 


25 


14 


2 


1 


Addition combinations, 10- 15 fl 
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lessons were short (the average length was less than 10 problems), and 
ions were simple and direct. Generally the problems within one 
lesson were all of the same type; the first few were accompanied by 
e xpl^ft^ory audio messages; the remainder were practice problems. 

The students received their programmed instruction in a room which 
contained 17 student stations and a proctor station for use by the teachers 
on duty. The student stations were separated by four-foot partitions that 
extended far enough from the walls to provide a degree of privacy for the 
students. When the children arrived in the student station room, they 
looked for their names on the CRT screen at their assigned stations, put 
on their headsets, and started their program by touching the light pen to 
a smiling face displayed on the CRT. After the allotted time for the class, 
approximately 20 minutes, the students were signed off automatically as 
they completed their current lesson and the message YOU HAVE BEEN SIGNED 
OFF appeared on the CRT. The children entered an adjacent classroom and 
joined a teacher who escorted them back to their homeroom. 

Both explanatory and practice problems contained provisional audio 
messages that were heard only by the students wha responded incorrectly ‘ 
or who failed to respond within a reasonable time. For example, for one 
problem, a drawing of a car and a drawing of a truck surrounded by set 
braces and followed by an equal sign was presented on the CRT; this problem 
was accompanied by the audio message "There are two members in this set." 
After this message, two more sets, one empty and one containing a train and 
a steamshovel, each preceded by a box, were displayed below the initial set; 
the choices were accompanied by the audio instructions, "Find another set 
with two members," At this point a small "p" (for pen) was displayed 
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in the corner of the CRT as a signal to the student to respond* If the 
student touched his light pen to the box in front of the correct choice, 
a smiling face was displayed and he heard, "Yes, the sets have the same 
number of members," and presentation of the next problem began* 

If the student did not respond within 20 seconds he heard, "Which 
set below has two members?" If the student responded incorrectly, he 
heard the audio message "Point to the box next to the set with two members," 
and saw a sad face* For most problems in the curriculum, students were 
allowed three chances to produce the correct answer* After three incorrect 
responses, a brief audio message accompanied the display of the correct 
answer or an arrow pointing to the correct choice* 

For most lessons the number of initial correct responses to the 
practice problems was accumulated and compared with a criterion* As soon 
as a student made the required number of correct responses, he was allowed 
to skip the remaining problems and to begin the next lesson. As soon as 
a student failed criterion, for instance, three incorrect responses if 
the criterion was seven out of nine problems, he was branched immediately 
to a remedial lesson containing the same kinds of problems, but with a 
slower development of ideas using simpler vocabulary and sentence structure. 
If a student failed criterion on a remedial lesson, his program stopped, 
and an automatic call for assistance from a proctor was typed at the proctor 
station. 

This mode of branching permitted students to progress through the 
curriculum at a rate consonant with their ability to master the concepts. 
Thus, all students in the class were not required to work on the same 
concept at the same time as in the drill- and-pr act ice program. In fact, 
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the faster students were separated considerably from the slower students 
in terms of curriculum material covered during the year. 

The classroom activity, completely coordinated with the programmed 
instruction, included (a) use of physical objects to introduce concepts 
presupposed by the programmed lessons; (b) work originally planned as 
programmed lessons; (c) remedial work for individual children; and (d) 
enrichment material for individual or group use. 

The curriculum presented during 1967-68 to the second graders contained 
lessons covering the topics of sets, numbers, relations, fractions, addition, 
subtraction, multiplication, division, connections between operations, 
geometry, measure, and problem solving. An outline of the programmed 
curriculum is shown in Table 8, 



Insert Table 8 about here 

f 



Observation of student behavior during the first year of the program 
(1966-67) indicated the advisability of several changes in the structure 
of the programmed lessons. Children in the first year who failed to respond 
to a problem within 20 seconds heard an additional audio message giving 
more detailed instructions and were then allowed additional time for a 
response. In the second-grade lessons (1967-68) an audio message, sometimes 
accompanied by visual clues, was presented to the student who requested 
help by touching the "help" button. This routine decreased the number and 
length of mandatory audio messages in the program. 

The remedial-lesson sequences were also changed. In the first-grade 
curriculum, remedial material was presented as a block of problems directly 
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TABLE 8 

Curriculum Outline for Grade 2 - Brentwood Tutorial Mathematics Program 
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linked to each lesson in the core curriculum. Thus, students who failed 
to meet criterion received immediate and detailed remedial instruction. 

In the second-grade curriculum, a pretest on every concept was given to 
each child. A child who met criterion on the pretest continued to other 
core material or to enrichment material, while a child who failed criterion 
did six to eight remedial lessons spaced over several days. 

Evaluation 

During the 1966-67 school year, 93 students were enrolled in the first 
grade at Brentwood. On the basis of reading scores achieved on an SAT 
administered in the fall, the students were grouped into four classes by 
the principal of the school. Two classes were at a low level on reading 
readiness and two classes were at a medium level on reading readiness. 
Forty-five of these children, one low-level class and one medium- level 
class, participated in the tutorial mathematics program. The other two 
classes, 48 children, participated in a computerized reading program and 
served as a control group for evaluation of the mathematics program. 

The Stanford-Binet I.Q., short form, was administered individually 
to 40 children in the experimental group and to 45 children in the control 
group during September 1966. The SAT Primary I Battery was administered 
to all of the first graders in May 1987 . The average I.Q. and the average 
grade placement on the SAT for the experimental and control groups and for 
the low- and medium- level classes within each group are shown in Table 9* 

Insert Table 9 about here 

Since the groups were equivalent in terms of I.Q. , the SAT grade -placement 
scores were appropriate for evaluation of differences between groups. 
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There was no significant difference between the total experimental group 
and the total control group, t(9l) = 1«10, p > .05, or between the medium- 
level experimental group and the medium- level control group, t(50) = .56, 
p > .05. The students in the low experimental group, however, performed 
significantly better than the students in the low control group, t ( 39) = 3*38, 
p < ,01. 

For the 1967-68 school year, all the second graders at Brentwood par- 
ticipated in the tutorial mathematics program. The second graders were 
divided into five classes--two high-level, two medium- level, and one low- 
level in terms of reading readiness. There were three classes in the control 
school — one high, one medium, and one low level. For evaluation, 66 second 
graders at Brentwood and 8l second graders at the control school, a school 
in the same district as Brentwood, were given the Primary I Battery of the 
SAT in the fall and the Primary II Battery in the spring. The average 
pretest and posttest grade placement for the experimental and control 
groups and for the low-, medium-, and high-level classes within each group 
are shown in Table 10, There was a significant difference between the two 

Insert Table 10 about here 

groups in terms of the change in the SAT score from fall to spring with 
the control students increasing their grade placement more than the experi- 
mental students, t(l 45 ) = - 2 , 83 ^ p < , 01 , In the analysis of the three 
subgroups, a significant difference was found for the high-level group, 
t(^ 5 ) = ** 5 « 03 j P < * 01 , but not for the medium- level group, t( 65 ) = -,55, 
p > .05, or the low-level group, t(3l) = .60, p > .05, 

Before these results can be interpreted, several facts should be 
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noted about the systems within which the testing and experimentation occurred. 
In both the first and second grades, the programmed curriculum differed 
from the curriculum taught in a first- or second-grade classroom, the 
difference being greater in the second grade. The differences between 
the two types of curriculum also might have been further decreased in the 
first grade since both the control and experimental subjects were from the 
same school. Thus, communication among teachers could have resulted in a 
change in curriculum for the control classes if the teachers of these classes 
had discussed their teaching plans with the teachers involved in the experi- 
mental program. This interaction between the two types of curriculums 
would be less likely to occur during the second year, since the two groups 
of subjects were from different schools. 

The differences in the two curriculums were such that the appropriate- 
ness of the SAT as an evaluatory test is questionable. The problems in 
the Computational Section of the Primary II Battery administered to the 
second grade were more similar to the material taught in the control school 
than to the material in the CAI curriculum, while much of the material 
included in the CAI curriculum did not appear on the SAT. A test based on 
the material taught in the computer-based classroom would not be a satis- 
factory alternative for evaluating the two teaching methods, unless the 
control school were presenting the same curriculum. The SAT results, 
however, proved useful for comparing the students at Brentwood with the 
students involved in the drill-and-practice program. 

Although the low-level class in the experimental first grade at 
Brentwood performed better on the posttest than the low-level class in 
the control school, the posttest scores on the SAT for both low-level 
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classes were below the performance level of the first-grade students tested 
in the spring of 1968 in the Mississippi and California experimental and 
control schools, (See Tables 5 and 6), Thus, the Brentwood program when 
compared with a control school program was successful for low-level students, 
but was not more successful than the drill-and-practice program when SAT 
scores were used as a performance measure. 

The second-grade program at Brentwood was not more effective than 
the control-school program for teaching the types of computations necessary 
to perform well on the SAT, In the high ability classes, the control stu- 
dents performed significantly better than the experimental students in 
terms of their fall to spring change in grade placement and in terms of 
the final posttest scores. Although the high-level control class was the 
only class to perform significantly better than the experimental classes, 
the posttest grade placement (2,9 6) was similar to the posttest grade 
placement of the second-grade control groups in California (2,90) and 
Mississippi (2,80), where the experimental groups were significantly better 
than the control groups. Unlike the situation reported for the drill-and- 
practice evaluation for 1966-67 where the superiority of the control group 
could be attributed to increased attention to mathematics in the control 
school (grades 4 and 5, Table 4), the superiority of the high-level control 
students in the Brentwood program, given their similarity to control students 
in the 1967-68 program, may well be attributed to a failure in the experi- 
mental program. Other problems experienced in the program are discussed 
in the final section of this paper. 
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3* The Russian Program 

The computer-based Russian program was instituted at Stanford in 
September 1967 under the direction of Professor Joseph Van Campen who 
designed a program to teach first- and second-year courses at the college 
level- This program included comprehension of written Russian, comprehension 
of spoken Russian, and mastery of grammar and syntax. Of the three main 
components of a college-level language course, i.e., classroom sessions 
on a daily basis, time in the language laboratory, and regular homework 
assignments, only the functions of the classroom sessions were assumed by 
the computer program. In addition to their time at the computer console, 
the students spent time in the language laboratory and did home assignments. 
The language-laboratory tapes with drill sheets and homework assignments 
are prepared by the staff at the Institute. 

First - year course , 1967-68 

Thirty students began the first-year Russian course in the fall of 
1967. Two of the four sections of beginning Russian served as a control 
group; the other two sections were asked to volunteer for the CAI course. 
None of the students refused to remain in their assigned sections. The 
CAI students were required to spend about 50 minutes a day, five days a 
week at the computer console. A total of 135 lessons were presented to 
the students in a combined audio and teletype format. The students re- 
sponded on a Model - 35 teletype with a special keyboard using the Cyrillic 
alphabet. 

Although the basic curriculum was the same for all students, there 
were several remedial branches. At given points in the curriculum, students 
were tested on several items of a given type and were given remedial 
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instruction on the material covered if their performance on the test block 
failed to meet a satisfactory standard. Later in the year (1967-68) routines 
were provided which produced more specific remedial work based on the type 
of error the student made. 

During the period prior to the final examination, lesson summaries 
for each new lesson and a final summary covering the material for the entire 
quarter were given to the students. The computer then assessed the student’s 
performance and told him the rules on which he should concentrate his efforts. 
At subsequent sessions the student was again tested on the material he had 
missed and was informed where more study was needed. In addition, the 
student could repeat any lesson or portion of a lesson at the computer 
console. 

Language-laboratory tapes provided material for pronunciation practice 
and also for testing a student’s ability to comprehend spoken Russian. A 
test at the end of the tape either required the student to transcribe into 
English a number of Russian sentences, or required the student to respond 
in writing to oral questions on a paragraph which he had just heard. 

In order to evaluate pronunciation, each student made two recordings 
during each quarter. After each recording session the student was counseled 
immediately and was told what pronunciation errors he had made and how to 
correct them. 

Second - year course , 1968-69 

Instruction began in September 1968 for 19 students enrolled in the 
second-year Russian course. Thirty-nine lessons, including review lessons, 
were available for the quarter. The students were at the console for about 
45 minutes, five days a week. Homework and study sheets for Lessons 1 
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through 39 were distributed to the students as they progressed through the 
lessons. The homework involved translating English sentences into Russian, 
while the study sheets dealt with new grammar and new vocabulary pertinent 
to the day's lesson. To develop speech and the ability to write correctly 
what was heard, the students occasionally were given an option at the con- 
clusion of a teletype lesson to take dictation or to practice pronunciation. 
These exercises were recorded at the end of the tape used as the audio 
portion of the regular teletype lessons. Once every two weeks students took 
written quizzes or read from handwritten or typed scripts. Their pronun- 
ciation was corrected and suggestions were made for improvement. 

... • . Evaluation 

Of the 30 students starting the first-year computer-based course, 1 left 
during the first quarter, 3 left between the first and second quarters, 1 left 
during the second quarter, and 3 left between the second and third quarters. 

Two new students entered the computer-based section at the beginning of the 
second quarter. Of the 38 students enrolled for the autumn quarter in the regu- 
lar Russian section, 10 left the course during the first quarter, 13 left between 
the first and second quarters, and 3 left between the second and third quarters. 
Four new students entered the regular section at the beginning of the third 
quarter, one of these transferred from the computer-based class. Of the 30 
students originally enrolled in the computer-based program, 22 (73 per cent) 
finished all three quarters, whereas of the 38 students in the regular class, 
only 12 (32 per cent) finished the year's curriculum. This finding suggests 
that the computer-based course held the interests of the students much better 
than did the regular course. Probably because Russian is more difficult 
than French, Spanish, or German for American students, the dropout rate 
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in Russian at Stanford and other universities is traditionally quite high. 
Approximately 66 per cent of the content of the final examinations 
for the autumn and winter quarters was identical for the computer-based 
and for the regular Russian sections; the complete final examination for 
the spring quarter was identical for the two groups. The error distribu- 
tion and the mean number of errors per student for the two groups on the 
final examination for the autumn, winter, and spring quarters are shown 
in Tables 11, 12, and 13, respectively. Although the average number of 

Insert Tables 11, 12, and 13 about here 

errors was lower for the computer-based students in all three quarters, 
the difference was actually statistically significant for the fall quarter 
(Mann-Whitney U Test, p < .001) and the spring quarter (p < .05), but not 
for the winter quarter. Since the selection process resulting from the 
poorer students leaving the regular course biases the results on the exam- 
ination against the computer-based group, the superiority of the computer- 
based group on the spring examination is more impressive than the difference 
indicated by the average number of errors. 

Of the 19 students enrolled in the second-year Russian course, 12 had 
participated in the first-year computer course, 7 took the Stanford place- 
ment test to qualify for the second-year course and were new fco computer- 
based instruction. Eleven students were enrolled in the second-year Russian 
course in the regular classroom. 

The distribution of errors for the computer-based class and the regular 
class on the final examination for the fall quarter is shown in Table 14. 

Insert Table 14 about here 
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TABLE 11 



Error Distribution for the Common Portion of the Autumn Quarter Final Examination, 

Russian Program 



Number of errors 


Number of students 


Computer-based 


Regular 


5-5 


1 




5 


2 


1 


6 


3 




7 


1 




8 


2 




9 


3 




11 


3 




15 




1 


15 


1 




16 


1 


1 


17 


2 




19 




1 


21 


2 


1 


22 


1 


1 


23 




2 


25 


1 


1 


27 


3 




29 




1 


30 




1 


31 




2 


33 


1 




34 




1 


37 


1 




38 


1 




4l 




1 


43 




1 


45 




1 


53 




1 


61 




1 



(continued) 
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Table 11 (continued) 

Number of errors 

64 

65 
72 
76 
79 
93 
97 

120 

l4l 



Number of students 



Computer-based 



Total number of students 
Average number of errors 



29* 

15.8 



Regular 



1 

1 

1 

1 

1 

1 

1 

1 

1 

± 



28** 

49.0 



* Of the 30 students enrolled, one left during the quarter. 
** Of the 38 students enrolled, ten left during the quarter. 




TABLE 12 



Error Distribution for the Common Portion of the Winter Quarter Final Examinat 

Russian Program 



Number of errors 


Number of students 


Computer-based 


Regular 


2 


1 


1 


6 


1 




6.5 


1 




8 


1 




9*5 




1 


10 


1 




n 




1 


12 


2 




13 




1 


1^*5 




1 


16 


1 




16.5 


1 




18 


1 




18.5 


1 




19 


1 




19*5 




1 


21 


2 




22.5 


1 


1 


23 


1 


1 


23*5 


1 




2k 


1 




24.5 




1 


25 


1 




26.5 




1 


27 


1 




29-5 


1 




30 




1 


30.5 




1 


32*5 




1 



(continued) 



Table 12 (continued) 



Number of errors 


Number of students 


Computer-based 


Regular 


33 


1 




37.5 


2 




38 


1 




39.5 


1 




4l 


1 


2 


47.5 




1 


Total number of students 


27* 


15** 


Average number of errors 


21.8 


25.8 



Three of the original students did not enroll, two new students were 
added, one student did not finish the quarter. 

** Thirteen of the original students did not enroll. 



46 





TABLE 13 



Error Distribution for the Spring Quarter Final Examination, 

Russian Program 





Table 13 (continued) 



Number of errors 


Number of students 


Computer-based 


Regular 


82 


1 




89 




1 


91 




1 


92 




1 


93 


1 




106 




1 


166 




1 


Total number of students 


24* 


16 ** 


Average number of errors 


53.0 


71.1 



* Three students did not enroll. 



** Three students did not enroll, three students enrolled for the first 
time, and one student transferred from the computer-based section to 
the regular section. 
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TABLE Ik 



Error Distribution for the Common Portion of the Fall Quarter Final Examination, 

Second-year Russian Program 



Number of errors 


Number of students 


Computer-based 


Regular 


2 


1 




3 


2 




4 


4 




5-5 


2 

1 




6 


1 




7 


1 




7-5 


2 




8.5 


1 




11 


1 


1 


11.5 


1 




12 




1 


13 




1 


15 




2 


15-5 


1 




16 


1 




16.5 


1 




17 




1 


17.5 




1 


19 




1 


21.5 




1 


22 




1 


24.5 




1 


Total number of students 19 


11 


Average number 


of errors 6.0 


15.7 

_[ 
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Seventy-four per cent of the canputer-based students performed better than 
the best student in the conventional class. The error distributions were 
significantly different (Mann-Whitney U Test, p < . 001 ); the computer-based 
students performed better on the examination than the conventional class- 
room students. 

Discussion of the Three Programs 

As is the case of any new technology being applied to an area where 
many existing skills and much existing knowledge is already present, the 
results of computer-assisted instruction in the early years will necessarily 
be mixed. From an operational standpoint, the Stanford efforts reflected 
in the three programs reported here began only in 1965, and so the evalua- 
tion reported here covers the first three years of effort. On balance, 
we feel that the results are positive, but it is also important to note 
that all the results have not been positive. We feel, however, that we 
have some explanation for some of the negative results. More importantly, 
we have learned a great deal since 1965, and we believe that the programs 
we are now developing and beginning to place in schools on an experimental 
basis will benefit from the work reported here. The technology is com- 
plicated, and our understanding of the underlying psychological principles 
of learning in any major area of curriculum is still rather tenuous. In 
our own judgment, while there is nothing definitive about any of the evalu- 
ation results reported in this paper, it did seem desirable to report as 
early as possible in systematic form and in objective fashion the evaluation 
results obtained. 
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We turn now to some specific remarks about the three programs. 

1. The most negative results, especially when evaluated with respect 
to the effort expended, were the relative achievement gains of the second- 
grade students at Brentwood in the tutorial mathematics program. Without 
being able to give a real causal explanation of these results, we feel that 
the central problem with the second-grade tutorial program in mathematics 
at Brentwood centered around operational difficulties. The curriculum 
material was more complex than that offered in the first year, and there were 
difficulties in debugging and operating the system satisfactorily. Perhaps 
the most important single variable was the unsatisfactory systems-response 
time, which was discouraging for the students. During the latter half of 
the second year, the response times on the system were often in excess of 

10 seconds. The technical reasons for these difficulties are now fairly 
well understood, but it is a warning to all computer-assisted instruction 
programs in elementary schools, and at other levels as well, that slow 
systems-response times are not tolerable. 

2. On the other hand, the concept of the tutorial approach at 
Brentwood is supported by the positive results with the slow learners in 
the first year. The problems of beginning learning with students who test 
in the I.Q. range of 65 to 90 upon entering school are difficult and ex- 
tensive. Teachers must have an infinite degree of patience to work with 
these students, to sense when progress is being made, and to repeat those 
things that need repeating. Tutorial programs that are computer-based 
provide one way of assisting teachers who are working with such classes. 

We should also like to say in connection with the tutorial program at 
Brentwood that this was the most radical program from the standpoint of 
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designing a full curriculum and the one whose parameters we as yet least 
understand. The Russian program also was tutorial and required a complete- 
ness of approach in the sense that almost the entire curriculum was handled 
at the computer. It is a simpler matter, however, to work with bright and 
mature college students than with younger relatively slow learners. 

3. Turning to the drill-and-practice program, we observe first that 
as the evidence comparing School A with School B in 1966-67 in California 
indicates, teachers can do as well with a good regime of drill and practice 
in the fundamentals of arithmetic as can computers. We do not think that 
this conclusion is at all surprising. We have known for a long time from 
studies dating back to the 20* s that a daily regime of drill and practice, 
carried out with faithfulness and regularity by the teacher, does improve 
the performance of students (see, for example, Wilson, 1925). What seems 
to be evident already is that the use of terminals to bring a drill-and- 
practice program to schools can bring a kind of quality control that is 
difficult to achieve in large numbers of schools with large numbers of 
teachers. Concentrated efforts in single schools with a dedicated staff 
can certainly do as well as anything that we can currently offer, but it 
is especially true of the elementary- school mathematics curriculum that 
many teachers in the upper three grades, that is, grades 4, and 6, are 
not really interested in mathematics and would much prefer to turn the 
problem of providing a regime of review and maintenance of arithmetic 
skills over to a computer-based instructional program. 

4. The results of the data reported here indicate that an individ- 
ualized drill-and-practice program in elementary mathematics will produce 
its most impressive results in school environments not educationally and 
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economically affluent. This is evident from the comparison of experimental 
and control groups in Mississippi and California. This remark is closely 
connected with the preceding one, for it is in the less affluent areas of 
the country that, in general, teacher preparation and teacher training are 
least satisfactory. One way to meet some of these problems of teacher 
training, as in mathemat '.cs, is to bring work to the student directly on 
computer-based terminals. Because there was no control school to match 
the change in achievement data that occurred in the spring of 1967, the 
evaluation data cited above did not include Elliotsville School in rural 
Kentucky. Striking effects can be achieved in deprived areas; for example, 
the average grade-placement increase for a fourth-grade class of 27 students 
was seven months after only one-and-a-half month’s work at teletype terminals. 

5* It would be a mistake, however, to conclude that it is only with 
deprived or slower students that computer-assisted instruction will show 
really effective results. The program in Russian at Stanford University 
provides clear evidence to the contrary. There is much about the teaching 
of foreign language that is particularly well suited to computer- assisted 
instruction. To keep pace with the programmed exercises, the student must 
concentrate more directly on the language and not return to an internal 
monologue in English as he listens to other students respond in a class 
of 20 or 30. The concentration required of the student at ccmputer-based 
terminals in the Russian program precludes his attention from wandering; 
thus, he achieves a degree of efficiency, it seems to us, that would be 
difficult to match, even in the best organized classroom. This is not to 
say that our Russian program is without defects. Professor Van Campen plans 
a large number of improvements for the program that will further deepen the 
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degree of individualization. We do feel that at both the secondary and 



college levels computer-based instruction can take over a good deal of 
the teaching of e foreign language, especially in those languages for 
w; *eh r't.rff is inadequate. From the standpoint of national interest, we 



the < t&ff for ^caching these three languages is not generally sufficient, 
particularly in secondary schools, 

6 . Another example not discussed here, but that provides clear evi- 



to the deprived or slower learners, is some of our work in logic and algebra 
at the elementary and beginning secondary- school levels. We have not pro- 
vided a classical evaluation of this program, which was one of our first 
curriculum efforts and began with demonstrations in December 19 63. There 
is no good direct comparative evaluation of control-group performance, 
since the body of curriculum material is not offered in ordinary classes. 
There is no doubt, however, that this program, which is primarily aimed 
at bright students in grades 4 to 8 , has been effective, because a great 
many mathematical ideas and skills have been learned by students who would 
not otherwise have been exposed to the material. Although the evidence 
is anecdotal, one of our finest examples is the rapid progress made in 
the logic program by students in Mississippi in comparison with students 
from upper middleclass environments in Palo Alto, We are especially proud 
of two Mississippi Negro boys in the eighth grade who stood at the top of 
the first-year logic program during 1967-68. This possibility of bringing 
enriched programs to students in a variety of environments where such 
courses cannot reasonably be offered by the teaching staff, either because 




instruction in Russian, Japanese, and Chinese, and yet 



dence that computer -assisted instruction is not restricted in its benefits 



5 ^ 
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of lack of time or lack of training, is probably one of the most imme- 
diately practical aspects of computer-assisted instruction. We want to 
conclude this evaluation paper by emphasizing the important role of such 
enrichment programs, and to emphasize their importance in spite of the fact 
that it is not easy to provide a classical, "hard data" evaluation of such 



programs. 



References 



Atkinson, R. C. Computerized instruction and the learning process, 
Amer. Psychol .. 1968, 25, 225-259. 

Kelley, T., Madden, R., Gardner, E. F., & Rudman, H. C. Stanford 

Ac hievement Test. New York: Harcourt, Brace & World, 1965. 

Suppes, P. Sets and numbers. Book 1. New York: Singer, 1965. 

Suppes, P„ German, M„ & Brian, D. Ccmputer- assisted instruction: 

— Stanford arithmetic program . New York: Academic 

Press., 1968. 

Suppes, P., & Morningstar, M. Computer-assisted instruction : fte 

1966-67 St anford arithmetic program . New York: Academic Press. 

1969^ in press. 



Wilson, G. M. Res earch in constructing the elementary school curriculum . 
Third yearbook of the Department of Superintendence. Washington, D, C 
National Education Association, 1925, 



Footnote 



The research reported here has been supported by the National 
Science Foundation and the U. S. Office of Education. The 
work has been a joint enterprise with many people. With respect 
to the dr ill-and -practice program, we are indebted especially 
to Mr, Max Jerman, IMSSS, and Dr. Barbara Searle, IMSSS, 

Mr. Julian Prince, Superintendent of Schools, McComb, Mississippi, 
and Mr. Edwin R, Jones, Director, Eastern Kentucky Educational 
Development Corporation, Ashland, Kentucky; with respect to the 
Brentwood tutorial mathematics program, M ' -Jamesine Friend, 
IMSSS, and Mr . William Rybensky, Brentwood Elementary School; 
and with respect to the Russian program, Professor Joseph 
Van Campen, IMSSS, and Mrs. Elise Belenky, IMSSS. 




(Continued from inside front cover) 



91 P. Suppes. Information processing and choice behavior. January 31, 1966. 

92 G. Groen and R. C. Atkinson. Models for optimizing the learning process. February II, 1966. ( Psychol . Bulletin, 1966, 66, 309-320) 

93 R. C. Atkinson and 0. Hansen. Computer-assisted instruction in Initial reading: Stanford project. March 17, 1966. ( Reading Research 

Quarterly , 1966, 2, 5-25) 

94 P. Suppes. Probabilistic inference and the concept of total evidence. March 23, 1966. (In J. Hintikka and P. Suppes (Eds.), Aspects of 

Inductive Logic. Amsterdam: North-Holland Publishing Co., 1966. Pp. 49-65. 

95 P. Suppes. The axiomatic method in high-school mathematics. April 12, 1966. (The Role of Axlomatlcs and Problem Solving In Mathematics^ 

The Conference 8oard of the Mathematical Sciences, Washington, 0. C. Ginn and Co., 1966. Pp. 69-76. 

96 R. C. Atkinson, J. W. Brelsford, and R. M. Shiffrin. Multi-process models for memory with applications to a continuous presentation task. 

April 13, 1966. (J. math. Ps ychol. , 1967, < , 277-300 ). 

97 P. Suppes and E. Crothers. Some remarks on stimulus-response theories of language learning. June 12, 1966. 

98 R. 8jork. All-or-none subprocesses in the learning of complex sequences. (J. math . Psychol. , 1968,^, 182-195). 

99 E. Ganmon. The statistical determination of linguistic units. July I, 1966. 

100 P. Suppes, L. Hyman, and M. Jerman. Linear structural models for response and latency performance In arithmetic, fln J. P. Hill (ed.), 

Minnesota Symposia on Child Psychology . Minneapolis, Minn.: 1967. Pp. 160-200). 

101 J. L. Young. Effects of intervals between reinforcements and test trials in paired-associate learning. August I, 1966. 

102 H. A. Wilson. An Investigation of linguistic unit size in memory processes. August 3, 1966. 

103 J. T. Townsend. Choice behavior in a cued-recognitlon task. August 8, 1966. 

104 W. H. 8atchelder. A mathematical analysis of multi-level verbal learning. August 9, 1966. 

105 H. A. Taylor. The observing response in a ctied psychophysical task. August 10, 1966. 

106 R. A. 8jork . Learning and short-term retention of paired associates In relation to specific i*quences of Interpresentation intervals. 

August II , 1966. 

107 R. C. Atkinson and R. M. Shiffrin. Some Two-process models for memory. September 30, 1966. 

108 P. Suppes and C. Ihrke. Accelerated program in elementary-school mathematics— the third year. January 30, 1967. 

109 P. Suppes and I. Rosenthal-Hill. Concept formation by kindergarten children in a card-sorting task. February 27, 1967. 

I 10 R. C. Atkinson and R. M. Shiffrin. Human memory: a proposed system and its control processes. March 21, 1967. 

I II Theodore S. Rodgers. Linguistic considerations in the design of the Stanford computer-based curriculum in initial reading. June I, 1967. 

I 12 Jack M. Knutson. Spelling drills using a computer-assisted instructional system. June 30, 1967. 

113 R. C. Atkinson. Instruction in initial reading under computer control: the Stanford Project. July 14, 1967. 

114 J. W. Brelsford, Jr. and R. C. Atkinson. Recall of paired-associates as a function of overt and covert rehearsal procedures. July 21, 1967. 

115 J. H. Stelzer. Some results concerning subjective probability structures with semiorders. August I, 1967 

116 D. E. Rumelhart. The effects of in ter presentation intervals on performance In a continuous paired -associate task. August II, 1967. 

I 17 E. J. Fishman, L. Keller, and R. E. Atkinson. Massed vs. distributed practice in computerized spelling drills. August 18, 1967. 

I 18 G. J. Groen. An Investigation of some counting algorithms for simple addition problems. August 21, 1967. 

I 19 H. A. Wilson and R. C. Atkinson. Computer-based instruction in Initial reading: a progress report on the Stanford Project. August 25, 1967. 

120 F. S. Roberts and P. Suppes. Some problems in the geometry of visual perception. August 31, 1967. ( Synthese, 1967, |7, 173-201) 

12 I D. Jamison. 8ayesian decisions under total and partial ignorance. 0. Jamison and J. Kozlelecki. Subjective probabilities under total 
uncertainty. September 4, 1967. 

122 R. C; Atkinson. Computerized instruction and the learning process. September 15, 1967. 

123 W. K. Estes. Outline of a theory of punishment. October I, 1967. 

124 T. S. Rodgers. Measuring vocabulary difficulty: An analysis of item variables in learning Russian-English and Japanese-English vocabulary 

parts. December 18, 1967. 

125 W. K. Estes. Reinforcement in human learning. December 20, 1967. 

126 G. L. Wolford, D. L. Wessel , W. K, Estes. Further evidence concerning scanning and sampling assumptions of visual detection 

models, January 31, 1968. 

127 R. C. Atkinson and R, M. Shiffrin. Some speculations on storage and retrieval processes in long-term memory. February 2, 1968. 

128 John Holmgren. Visual detection with imperfect recognition. March 29, 1968. 

129 Lucille 8. Mlodnosky, The Frostig and the 8ender Gestalt as predictors of reading achievement, April 12 , 1968. 

130 P, Suppes. Some theoretical models for mathematics learning. April 15, 1968. (Journal of Research and Development in Education . 
1967, l § 5-22) 

131 G. M. Olson. Learning and retention in a continuous recognition task. May 15, 1968, 

132 Ruth Norene Hartley. An investigation of list types and cues to facilitate Initial reading vocabulary acquisition. May 29, 1968. 

133 P, Suppes. Stimulus-response theory of finite automata. June 19, 1968. 

134 N. Moler and P. Suppes, Quantifier-free axioms for constructive plane geometry. June 20, 1968. (In J. C. H. Gerretsen and 

F, Oort (Eds.), Compositio Mathematica . Vol. 20. Groningen, The Netherlands: Wolters-Noordhoff, 1968. Pp. 143-152.) 

135 W, K. Estes and D. ?. Horst. Latency as a function of number or response alternatives in paired-associate learning. July I, 1968. 

136 M, Schlag-Rey and P, Suppes, High-order dimensions in concept Identification. July 2, 1968. (Psychom , Sci ., 1968, N, 141-142) 

137 R. M. Shiffrin. Search and retrieval processes in long-term memory. August 15, 1968. 

138 R. D. Freund, G. R. Loftus, and R.C. Atkinson. Applications of multiprocess models for memory to continuous recognition tasks. 

December 18, 1968. 

139 R. C. Atkinson. Information delay in human learning. December 18, 1968. 

140 R. C. Atkinson, J. E. Holmgren, and J. F. Juola. Processing time as influenced by the number of elements in the visual display. 

March 14, 1969. 

141 P. Suppes, E. F. Loftus, M. Jerman. Problem-solving on a computer-based teletype. March 25, 1969. 

142 P. Suppes and Mona Morningstar. Evaluation of three computer-assisted instruction programs. May 2, 1969. 




